Exploratory data analysis

Table of Contents

• Data import
• Identify drop features
• Visualizing features

# Import libraries
import re
import sys
from hashlib import sha1
from pandas_profiling import ProfileReport
import altair as alt

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression

# train test split and cross validation
from sklearn.model_selection import (
    train_test_split,
)

Data import

---

imdb_df = pd.read_csv("https://raw.githubusercontent.com/nproellochs/SentimentDictionaries/master/Dataset_IMDB.csv")
imdb_df

	Id	Text	Author	Rating
0	29420	In my opinion, a movie reviewer's most importa...	Schwartz	0.1
1	17219	STARSHIP TROOPERS (director: Paul Verhoeven; c...	Schwartz	0.2
2	18406	THE SCHOOL OF FLESH (ECOLE DE lA CHAIR, L') (D...	Schwartz	0.2
3	18648	LOCK, STOCK AND TWO SMOKING BARRELS (director:...	Schwartz	0.2
4	20021	RUN LOLA RUN (LOLA RENNT)(director/writer: Tom...	Schwartz	0.2
...	...	...	...	...
5001	7470	The conventional wisdom is that movie sequels ...	Rhodes	0.9
5002	7853	Nicolas Roeg's mesmerizing 1971 film WALKABOUT...	Rhodes	0.9
5003	8309	The movie AIR FORCE ONE should require a docto...	Rhodes	0.9
5004	8912	"Well, Jones, at least you haven't forgotten h...	Rhodes	0.9
5005	9085	In a time of bloated productions where special...	Rhodes	0.9

5006 rows × 4 columns

Split the data into train_df (80%) and test_df (20%).

train_df, test_df = train_test_split(imdb_df, test_size=0.2, random_state=123)

Data overview

---

profile = ProfileReport(train_df, title='Pandas Profiling Report') #, minimal=True)
profile.to_notebook_iframe()

pd.DataFrame(train_df['Rating'].value_counts())

	Rating
0.70	688
0.50	640
0.60	637
0.40	484
0.80	387
...	...
0.14	1
0.32	1
0.22	1
0.17	1
0.99	1

86 rows × 1 columns

train_df.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 4004 entries, 4715 to 3582
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   Id      4004 non-null   int64  
 1   Text    4004 non-null   object 
 2   Author  4004 non-null   object 
 3   Rating  4004 non-null   float64
dtypes: float64(1), int64(1), object(2)
memory usage: 156.4+ KB

Data Format

Column Name	Column Type	Description
Id	Numeric	Unique ID assigned to each observation.
Text	Free Text	Body of the review content.
Author	Categorical	Author's name of the review
Rating	Numeric	Ratings given along with the review

Identify drop features

---

drop_features = ['id', 'Author']

Visualizing features

---

train_df.describe(include="all")

	Id	Text	Author	Rating
count	4004.000000	4004	4004	4004.000000
unique	NaN	3978	4	NaN
top	NaN	Gordon Comstock, advertising copywriter par ex...	Rhodes	NaN
freq	NaN	2	1415	NaN
mean	13887.965035	NaN	NaN	0.580564
std	9332.642525	NaN	NaN	0.181505
min	1858.000000	NaN	NaN	0.000000
25%	5404.250000	NaN	NaN	0.450000
50%	11211.000000	NaN	NaN	0.600000
75%	23263.250000	NaN	NaN	0.700000
max	29866.000000	NaN	NaN	1.000000

The distribution of ratings

alt.Chart(train_df).mark_bar().encode(
    x=alt.X('Rating', bin=True, title='Rating'),
    y='count()')

The most of ratings are from 0.4 to 0.9. There is some imbalance for our training dataset.

The list of top 20 frequent word appeared in text

vectorizer = CountVectorizer()
imdb_text_counts = vectorizer.fit_transform(train_df['Text'])

pd.DataFrame(data = imdb_text_counts.sum(axis=0).tolist()[0], 
             index = vectorizer.get_feature_names(), columns=['counts']) \
    .sort_values('counts', ascending=False) \
    .head(20)

	counts
the	172557
of	78038
and	76392
to	74239
is	57547
in	49646
that	33476
it	33061
as	27536
with	26852
he	26320
for	25704
his	25454
on	18487
film	17220
by	16890
this	16778
but	16759
her	15690
are	15084

From the above table, we could easily find that the top 20 words are meaningless to do the prediction. When we build our model we need to take it into account to ignore these words.

The relationship between text length and ratings

str_len = train_df['Text'].str.len()
train_df.loc[:, 'TextLength'] = str_len

alt.Chart(train_df, title= 'Text length vs ratings').mark_bar().encode(
    x=alt.X('Rating', bin=True, title='Rating'),
    y=alt.Y('mean(TextLength):Q', title='Mean of the text length'))

/Users/marcoma/opt/miniconda3/envs/573/lib/python3.8/site-packages/pandas/core/indexing.py:1596: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self.obj[key] = _infer_fill_value(value)
/Users/marcoma/opt/miniconda3/envs/573/lib/python3.8/site-packages/pandas/core/indexing.py:1743: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  isetter(ilocs[0], value)

From the graph, we can see the text length has no clear relationship with the rating, the average of len is around from 4000 to 5000. Except 0.9-1.0, its length is over 5,500.